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ABSTRACT: 


The pervasive sensing technologies found in smart homes offer unprecedented opportunities for providing 
health monitoring and assistance to individuals experiencing difficulties living independently at home. A prima- 
ry challenge that needs to be tackled to meet this need is the ability to recognize and track functional activities 
that people perform in their own homes and everyday settings. In this paper we look at approaches to perform 
real-time recognition of Activities of Daily Living. We enhance other related research efforts to develop ap- 
proaches that are effective when activities are interrupted and interleaved. To evaluate the accuracy of our 
recognition algorithms we assess them using real data collected from participants performing activities in our 


on-campus smart apartment testbed. 
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INTRODUCTION 


A convergence of technologies in machine learning 
and pervasive computing has caused interest in the 
development of smart environments to emerge and 
assist with valuable functions such as remote health 
monitoring and intervention. An estimated 9% of 
adults age 65+ and 50% of adults age 85+ need as- 
sistance with everyday activities, and the resulting 
cost for governments and families is daunting. The 
resulting need for development of such technologies 
is underscored by the aging of the population, the 
cost of formal health care, and the importance that 
individuals place on remaining independent in their 
own homes. When surveyed about assistive technol- 
ogies, family caregivers of Alzheimer’s patients 
ranked activity identification, functional assessment, 
medication monitoring and tracking at the top of 
their list of needs [1]. 


To function independently at home, individuals need 
to be able to complete both basic (e.g., eating, dress- 
ing) and more complex (e.g., food preparation, 
medication management, telephone use) Activities 
of Daily Living (ADLs) [2]. Smart environments 
can play an assistive role in this context. Our long- 
term goal is to design smart environment technolo- 
gies that monitor the functional well-being of 
residents and provide assistance to help them live 
independent lives in their own homes. The goal of 


this project is to design an algorithm that labels the 
activity that an inhabitant is performing in a smart 
environment based on the sensor data that is col- 
lected by the environment during the activity. In the 
current study, our goal is to design and test various 
probabilistic modeling methods that can recognize 
activities in more complex situations when activities 
are interrupted and interwoven. To test our ap- 
proach, we collect sensor data in our smart 
apartment testbed while participants perform activi- 
ties sequentially and interwoven together. We use 
this collected data to assess the recognition accuracy 
of our algorithms. 


There is a growing interest in designing smart envi- 
ronments that reason about residents [3,4], provide 
health assistance [5], and perform activity recogni- 
tion [6,7,8]. However, several challenges need to be 
addressed before smart environment technologies 
can be deployed for health monitoring. These in- 
clude the design of activity recognition algorithms 
that generalize over multiple individuals and that 
operate robustly in real-world situations where ac- 
tivities are interrupted. This technology, if accurate, 
can be used to track activities that people perform in 
their everyday settings and to remotely and automat- 
ically assess their functional well-being. 
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sensor ID 


date / time 


| reading 


12048 146000000B2 | 


2008-02-12 10:50:45.673225 | ON 


12D27E460000000D | 


2008-02-12 10:50:48.903745 | ON 


12048 146000000B2 | 


2008-02-12 10:50:49.339849 | OFF 


2084A30D00000039B | 


2008-02-12 10:50:53.27364 1 0.0459382 


2084A30D00000039B | 


2008-02-12 10:51:05.6252 10.158401 


Fig. 1 Resident performing “hand washing” activity (left). This activity triggers motion sensor ON/OFF events as well as water 
flow sensor values (right). Sensors in the apartment (bottom) monitor motion (M), temperature (T), water (W), door (D), burner 
(AD), and item use (I). 
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METHODS 


Data Collection 


To validate our algorithms, we test them in a smart 
apartment testbed located on the WSU campus. The 
testbed is equipped with motion and temperature 
sensors as well as analog sensors that monitor water 
and stove burner use (see Figure 1). The motion 
sensors are located on the ceiling approximately 1 
meter apart and are focused to provide 1 meter loca- 
tion resolution for the resident. Voice over IP 
(VOIP) technology captures phone usage and we 
use contact switch sensors to monitor usage of the 
phone book, a cooking pot, and the medicine con- 
tainer. Sensor data is captured using a customized 
sensor network and stored in a SQL database. 


To provide physical training data for our algorithms, 
we recruited 20 volunteer participants to perform a 
series of activities in the smart apartment, one at a 
time. For this study, we selected 8 ADLs that are 
found in clinical questionnaires [9]. Noted difficulties 
in these areas can help identify individuals who may 
be having trouble functioning independently at home 
[10]. These activities are as follows: 


1. Fill medication dispenser: Here the pattici- 
pant removes the items from kitchen 
cupboard and fills the medication dispenser 
using the space on the kitchen counter. 


2. Watch DVD: The participant selects the 
DVD labeled “Good Morning America” lo- 
cated on the shelf below the TV and watches 
it on the TV. After watching it, the partici- 
pant turns off the TV and returns the DVD 
to the shelf. 


3. Water plants: For this activity, the partici- 
pant takes the watering can from the supply 
closet and lightly waters the 3 apartment 
plants, 2 of which are located on the kitchen 
windowsill and the third is located on the 
living room table. After finishing, he/she 
empties any extra water from the watering 
can into the sink and returns the watering 
can to the supply closet. 


4. Converse on Phone: Here the participant an- 
swers the phone when it rings and hangs up 
after finishing the conversation. The con- 
versation includes several questions about 
the DVD show that the participant watched 
as part of activity 2. 


ISSN: 0975 — 2161 
IJBSPT (2009), 1(1):25-35 


5. Write Birthday Card: The participant writes 
a birthday wish inside the birthday card and 
fills out a check in a suitable amount for a 
birthday gift, using the supplies located on 
the dining room table. He/she then places 
the card and the check in an envelope and 
appropriately addresses the envelope. 


6. Prepare meal: The participant uses the sup- 
plies located in the kitchen cupboard to 
prepare a cup of noodle soup according to 
the directions on the noodle soup package. 
He/she also fills a glass with water using the 
pitcher of water located on the top shelf of 
the refrigerator. 


7. Sweep and dust: For this task, the participant 
sweeps the kitchen floor and dusts the din- 
ing and the living room using the supplies 
located in the kitchen closet. 


8. Select an outfit: Lastly, the participant se- 
lects an outfit from the clothes closet to be 
worn by a male friend going on an important 
job interview. He/she then lays out the se- 
lected clothes on the living room couch. 


We first asked the participants to perform each activi- 
ty separately, one at a time. [Each participant 
performed the separated activities in the same sequen- 
tial order. Once they completed all eight activities, 
the participants were then instructed to perform all of 
the activities again. This time they were allowed to 
interweave them in any fashion they liked with a goal 
of being efficient in performing the tasks. 


As part of the initial activity collection, we obtained 
sensor events for each activity that could be used to 
generate a model of each individual task. During the 
second phase in which participants interwove the 
tasks, the order in which activities were performed 
and were interwoven was left to the discretion of the 
participant. Because different participants interwove 
the tasks differently, the resulting data set was much 
richer and more complex than the first. Many partic- 
ipants performed several tasks in parallel. For 
example, one individual watched the DVD, talked on 
the phone, and watered plants all at the same time. 


The data collected during these tasks was manually 
annotated with the corresponding activity for model 
training purposes. Specifically, each sensor event 
was labeled with the corresponding activity id. The 
average times taken by the participants to complete 
the eight activities were 3.5 minutes, 7 minutes, 1.5 
minutes, 2 minutes, 4 minutes, 5.5 minutes, 4 minutes 
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and 1.5 minutes, respectively. The average number 
of sensor events collected for each activity was 31, 
59, 71, 31, 56, 96, 118, and 34, respectively. 


Probabilistic Models 


While collecting sequences of sensor readings in a 
smart environment is valuable, determining what 
activities these sequences represent is a more chal- 
lenging task. Researchers [11,12] frequently exploit 
probabilistic models to recognize activities and 
detect anomalies in support of individuals living at 
home with special needs. Most of the activity rec- 
ognition research that has been conducted to date 
focuses on recognizing activities in artificial scena- 
rios and when activities are performed sequentially. 
In contrast, we are focusing on recognizing activi- 
ties in realistic situations when activities are 
interrupted, performed in parallel and interleaved 
together. 


In our study, we make use of three probabilistic 
models to represent and recognize activities based 
on observed sensor sequences: a naive Bayes clas- 
sifier, a Markov chain model, and a hidden Markov 
model. We later describe enhancements that are 
employed to improve the recognition accuracy and 
test the algorithms on the data collected in the smart 
apartment testbed. 


Naive Bayes Classifier 


A naive Bayes classifier uses the relative frequen- 
cies of feature values and the activity labels for the 
sample training data to learn a mapping from a data 
point description to a classification label. For our 
application, activities are represented by features 
including the number of times during the activity 
that the water or burner was on/off, whether or not 
the phone was used, the number of times doors or 
cabinets were open/shut, whether items of interest 
were used, and the number of times the resident was 
at each location. 


The activity label, A, is _ calculated as 
P(D\a)P(a 
argmax,., P(al| D) = PONE) In this 
P(D) 


calculation D represents the feature values. The 
denominator will be the same for all values of a so 
we calculate only the numerator values, for which 
P(a) is estimated by the proportion of cases for 
which the activity label is a (in our case each partic- 
ipant performed all five activities so there is a 
uniform probability over all activity values) and 
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P(Dla) is calculated as the probability of the feature 
value combination for the particular observed activi- 


ty, or II, P(d; la). 


Markov Model 


A Markov Model (MM) is a statistical model of a 
dynamic system. A MM models the system using a 
finite set of states, each of which is associated with a 
multidimensional probability distribution over a set 
of parameters. The parameters for the model are the 
feature values described above. The system is as- 
sumed to be a Markov process, so the current state 
depends on a finite history of previous states (in our 
case, the current state depends only on the previous 
state). Transitions between states are governed by 
transition probabilities.For any given state a set of 
observations can be generated according to the asso- 
ciated probability distribution. Because our goal is 
to identify the activity that corresponds to a se- 
quence of observed sensor events, we generate one 
Markov model for each activity that we are learning. 
We use the training data to learn the transition prob- 
abilities between states for the corresponding 
activity model and to learn probability distributions 
for the feature values of each state in the model. 


To label a sequence of sensor event observations 
with the corresponding activity, we compute the 
activity label A using Bayes formula as 


argmax ,., P(ale, ,)=P(e,, |a)P(a). Pa) 


is estimated as before, while P(e, , | @) is the re- 


acA 


sult of computing the sum, over all states, S, in 
model a, of the likelihood of being in each state 
after processing the sequence of sensor events e7,;. 
The likelihood of being in state se Sis updated 
after each sensor event e; is processed using the 
formula found in Equation 1. The probability is 
updated based on the probability of transitioning 
from any previous state to the current state (the first 
term of the summation) and the probability of being 
in the previous state given the sensor event sequence 
that led up to event e,. 


1 
P(S, le = “ 
P(e 1S))D), POS; 1 5..)P 414.54) 


In constructing the model for an activity, we treat 
each sensor as a separate state with a corresponding 
probability distribution over possible sensor values. 
An example Markov model for this application is 
shown in Figure 2. Using this model, the sensor 
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Fig. 2 Markov model representing the "washing hands" activity. 


event sequence [Motion 14, Motion 15, Motion 16, 
Motion 17, Water On, Water Off] would be sup- 
ported with a probability that is calculated as 
follows. 


First, we obtain the prior probability of all states for 
each activity. In this case, we calculate the prior 
probability of a state as the ratio of the number of 
sensor events representing the state in an activity to 
the total number of sensor events recorded for that 
activity. 


Next, we calculate the probability of transitioning 
from a previous state to possible alternatives for the 
current state. Transition probabilities are calculated 
as the ratio of the number of transitions that are ob- 
served in the training data set from one state to the 
next to the total number of sensor events that are 
observed for the state. 


In the next step, we calculate the likelihood of being 
in each possible state for each possible activity. 
This is calculated as the product of the prior proba- 
bility of the initial state and the transition 
probability of moving from the initial state to the 
final state. This value is then multiplied by the 
probability of observing the current sensor value for 
the given activity. 


Finally, the probability that a model supports the 
given sensor sequence is calculated for each model 
(activity) as the sum of the likelihood values calcu- 
lated for each state in that activity. As an example, 


the probability that the “washing hands” model in 
Figure 2 supports the sequence [Motion 14, Motion 
15, Motion 16, Motion 17, Water On, Water Off] is 
0.1. Because this probability is greater than the 
probability values generated by the models for cook- 
ing, making a phone call, or cleaning up, the 
washing hands activity is the label that is output for 
this input sensor event sequence. 


One enhancement we make to a basic Markov mod- 
el is to incorporate the relative timing of each sensor 
event. In order to make use of the timing informa- 
tion we add a distribution over the relative times 
between events to the state feature distributions for 
each state in the model. 


Hidden Markov Model 


In earlier work [13] we showed that Markov chains 
were effective at recognizing activities performed in 
isolation. They could also be used to assess how 
completely activities were performed. For inter- 
leaved data, however, a Markov chain would not 
always be appropriate because we will not be able to 
separate the sensor event sequence into non- 
overlapping subsequences that correspond to each 
individual activity. In fact, some sensor events cor- 
respond to more than one activity in cases where the 
participant performed more than one activity in paral- 
lel. 


A hidden Markov model (HMM) is a statistical mod- 
el in which the underlying model is a stochastic 
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process that is not observable (i.e. hidden) and is as- 
sumed to be a Markov process which can be 
observed through another set of stochastic processes 
that produce the sequence of observed symbols. A 
HMM assigns probability values over a potentially 
infinite number of sequences. But as the probabili- 
ties values must sum to one, the distribution 
described by the HMM is constrained. This means 
that the increase in probability values of one se- 
quence is directly related to the decrease in 
probability values for another sequence. In the case 
of a Markov chain, all states are observable states and 
are directly visible to the observer. Thus, the only 
other parameter in addition to the prior probabilities 
of the states and the distribution of feature values for 
each state is the state transition probabilities. In the 
case of a hidden Markov model, there are hidden 
states which are not directly visible, and the observa- 
ble states (or the variables) influence the hidden 
states. Each state is associated with a probability dis- 
tribution over the possible output tokens. Transitions 
from any one state to another are governed by transi- 
tion probabilities as in the Markov chain. Thus, in a 
particular state an outcome can be generated accord- 
ing to the associated probability distribution. 


HMMs are known to perform very well in cases 
where temporal patterns need to be recognized 
which aligns with our requirement in recognizing 
interleaved activities. The conditional probability 
distribution of any hidden state depends only on the 
value of the preceding hidden state. The value of an 
observable state depends only on the value of the 
current hidden state. 
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Given a set of training data our algorithm uses the 
sensor values as parameters of the hidden Markov 
model. Given an input sequence of sensor event 
observations, our goal is to find the most likely se- 
quence of hidden states, or activities, which could 
have generated the observed event sequence. We 
use the Viterbi algorithm [14] to identify this se- 
quence of hidden states. In our implementation of a 
hidden Markov model, we treat every activity as a 
hidden state. As a result, our HMM includes 8 hid- 
den states, each of which denotes one of the 8 
modeled activities. Next, every sensor is treated as 
an observable state in the model due to the fact that 
every sensor which is used is observable in our data- 
set. The challenge here is to identify the sequence of 
activities (i.e., the sequence of visited hidden states) 
that corresponds to a sequence of sensor events (i.e., 
the observable states). For this, we calculate based 
on the collected data the prior probability (i.e., the 
start probability) of every state which represents the 
belief about which state the HMM is in when the 
first sensor event is seen. For a state (activity) a, this 
is calculated as the ratio of instances for which the 
activity label is a. We also calculate the transition 
probability which represents the change of the state 
in the underlying Markov model. For any two states 
a and b, the probability of transitioning from state a 
to state b is calculated as the ratio of instances hav- 
ing activity label a followed activity label b, to the 
total number of instances. The transition probability 
signifies the likelihood of transitioning from a given 
state to any other state in the model and captures the 
temporal relationship between the states. 


Fig. 3 A section of an HMM for interleaved activity data. The circles represent hidden states (i.e., activities) and the rectangles 
represent observable states. Values on horizontal edges represent transition probabilities and values on vertical edges represent the 
emission probability of the corresponding observed state. 


Lastly, the emission probability represents the like- 
lihood of observing a particular sensor event for a 
given activity. This is calculated by finding the fre- 
quency of every sensor event as observed for each 
activity. Using the Markov chain, a separate model 
was generated for each alternative activity. In con- 
trast, we only generate one HMM to represent all 
activities. Whereas the Markov chain would match 
the sensor event sequence with each model and re- 
turn the label of the model that best supports the 
sequence, the HMM process the event sequence as a 
continuous stream of data and the activity (hidden 
node) with the highest probability is returned as the 
activity label corresponding to the most recent sen- 
sor event. Figure 3 shows a portion of an HMM for 
interleaved activities. 


RESULTS 
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Fig. 4 Performance of naive Bayes classifier broken down by 
activity. Each bar shows the percentage of incorrectly-classified 
instances (bottom) and correctly-classified instances (top). 


We used data collected in our smart apartment 
testbed to train the naive Bayes classifier and HMM. 
The naive Bayes classifier achieved an average rec- 
ognition accuracy of 66.08%. The accuracy values 
for each activity are graphed in Figure 4. As the 
figure shows, activities “water plants” and “con- 
verse on phone” yield the poorest performance. The 
water plants activity is associated with almost ran- 
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dom movements around the apartment, while no 
sensor is consistently triggered while the participant 
talks on the telephone: some participants sit in one 
spot while others walk around the apartment as they 
talk. As a result, there is no distinct and consistent 
set of sensors values associated with these two ac- 
tivities. Because a hidden Markov model encodes 
sequencing information between states, we expect 
that it will outperform the naive Bayes classifier. In 
fact, on this dataset the HMM achieves an average 
recognition accuracy of 71.01%, which represents a 
significant improvement of 5% accuracy over the 
naive Bayes model at p<0.04. Figure 5 graphs the 
accuracy values for each activity. A limitation of 
the HMM is that because it processes sensor events 
in a stream, activities may in reality transition much 
faster than the probabilities transition from one ac- 
tivity to another. When the participant starts a new 
activity, several sensor events must occur before the 
probability indicates that the new activity has started 
as opposed to continuing the old one. Because we 
are labeling each sensor event with an activity label 
based on the sensor event sequence that preceded 
the current reading, this slow transition affects the 
performance of this HMM. 
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Fig. 5 Performance of HMM broken down by activity. 


To address this issue, we introduce an enhancement 
to the HMM which employs a sliding window. A 
window of fixed size moves over the sensor data 
stream. Only sensor events within the window are 
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input to the HMM to determine the current activity. 
The size of the window can be based on time or on 
number of sensor events. Another challenge that 
arises in using a sliding window is determining the 
appropriate window size. The optimal window size 
may change for different activity recognition appli- 
cations. For this reason we want to automate 
window size selection based on features of the cur- 
rent dataset. In order to automate window size 
selection, our algorithm automatically experiments 
with different window sizes on the training data. 
The window size that achieves the best results, using 
3-fold cross validation on the training data, is used 
for the remainder of the activity recognition process. 
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Number of sensor events 
Fig. 6 Accuracy of model for alternative count-based window 
sizes. 


The graph in Figure 6 shows the accuracy of the 
HMM for various count-based window sizes. Note 
that the performance increases as the window size 
increases because more context information is used 
to determine the current activity. Performance 
peaks at a window size of 57 sensor events, which is 
the size that the algorithm thus uses for the activity 
recognition. Performance starts falling again when 
the window size is too large, likely larger than the 
number of events that typically occur for each activ- 
ity. In addition to applying a sliding window, we 
also changed our activity labeling approach. Instead 
of labeling each sensor event with the most probable 
activity label, we now determine the activity label 
for the entire window. We can then label the last 
sensor event in the window with the activity label 
that appears most often in the window (a frequency 
approach) and more the window down the stream by 
one event to label the next event. Alternatively, we 
can label all sensor events in the window with the 
activity label that most strongly supports the se- 
quence and then shift the window to cover a non- 
overlapping set of new sensor events in the stream 
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(a shifting window approach). Figure 7 compares 
the performance of all of the discussed techniques. 


DISCUSSION 


The goal of this project was to design an algorithmic 
approach to recognize activities performed in a real- 
time, complex, smart environment. Our experimen- 
tal results indicate that it is possible to distinguish 
between activities that are performed in a smart 
home and to label a sensor event stream with high 
accuracy. When a probabilistic model is used, the 
approach can be used to recognize activities even 
when they are interwoven. We note that overall the 
hidden Markov models outperformed the naive 
Bayes classifier on the activity recognition task. 
This is due primarily to the fact that the Markov 
models represent and include sequencing informa- 
tion when they compute the likelihood of the 
activity sequence that corresponds to the observed 
sensor sequence. 


Among the hidden Markov model variations, the 
models that moved a window over the event stream 
outperformed those that did not. There was no clear 
winner between time-based windows and count- 
based windows. The best approach in determining 
the size and measure for the sliding window is to 
experiment with different values on training data 
and automatically select the one with the best per- 
formance for use on the remainder of the dataset. 
The approach that performed best was the hidden 
Markov model that used a time-based shifting win- 
dow to label all activities inside the window with the 
same activity name. Figure 8 shows the accuracy of 
both shifting window approaches broken down by 
ADL activity. In all of the experiments, the accura- 
cy level varied widely by activity. For the time- 
based shifting window HMM, the poorest accuracy 
was realized for the “water plants’ activity 
(28.45%), while the maximum accuracy was rea- 
lized for the “sweep and dust” activity (88.14%). 
This highlights the fact that smart environment algo- 
rithms need to not only performed automated 
activity recognition and tracking, but they need to 
base subsequent responses on the recognition accu- 
racy that is expected for a particular activity. If the 
home intends to report to the caregiver changes in 
the performance activity, such changes should only 
be reported for activities where recognizing and 
assessing the completeness of the activity can be 
accomplished with consistent success. 
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Comparison of different techniques 
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Fig. 7 Performance comparison of all approaches in recognized interleaved activities. 
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Fig. 8 Performance comparison of the time-based and count-based shifting window approaches for the HMM, broken down by activity. 
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While our study revealed that Markov models are 
effective tools for recognizing activities, there are 
even more complex monitoring situations that we 
need to consider before the technology can be dep- 
loyed. In particular, we need to design algorithms 
that perform accurate activity recognition and track- 
ing for environments that house multiple residents. 
In addition, we need to design method for detecting 
errors in activity performance and for determining 
the criticality of detected errors. 


In our data collection, an experimenter informed the 
participants of each activity to perform. In more rea- 
listic settings, such labeled training data will not be 
readily available and we need to research effective 
mechanisms for training our models without relying 
upon excessive input from the user. We hypothesize 
that ADL recognition and assessment can be per- 
formed in such situations and our future studies will 
evaluate the ADL recognition and assessment algo- 
rithms in actual homes of volunteer participants. 


CONCLUSIONS 


In this work we described an approach to recognizing 
activities performed by smart home residents. In par- 
ticular, we designed and assessed several algorithms 
that built probabilistic models of activities and used 
them to recognize activities in complex situations 
where residents switched frequently between the ac- 
tivities. 


Ultimately, we want to use our algorithm design as a 
component of a complete system that performs func- 
tional assessment of adults in their everyday 
environments. This type of automated assessment 
also provides a mechanism for evaluating the effec- 
tiveness of alternative health interventions. We 
believe these activity profiling technologies are valu- 
able for providing automated health monitoring and 
assistance in an individual’s everyday environments. 
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